1. Field of the Invention
The present invention relates to an image processing device and an image processing method that enable to preserve an edge in a high-quality image.
2. Description of the Background Art
In recent years, in an image sensing apparatus such as a digital camera, as a high-quality image is demanded, a technology of expanding a luminance range i.e. a dynamic range of a subject image to be captured by an image sensor is required. In expanding the dynamic range, there is known an image sensor (hereinafter, called as “linear-logarithmic sensor”) having a photoelectric conversion characteristic (hereinafter, also called as “linear/logarithmic characteristic”) comprised of a linear characteristic and a logarithmic characteristic. An image which is captured by the linear-logarithmic sensor, and has the linear/logarithmic characteristic is called a “linear-logarithmic image”. The linear-logarithmic sensor is capable of generating a natural logarithmically converted output with respect to an incident light amount in the logarithmic characteristic. Therefore, the linear-logarithmic sensor is advantageous in securing a wide dynamic range, as compared with an image sensor merely having a linear characteristic as the photoelectric conversion characteristic.
Observing the linear/logarithmic characteristic of the linear-logarithmic sensor, the linear-logarithmic sensor is capable of generating a high contrast image by photoelectric conversion in the linear characteristic, because the linear characteristic of the linear-logarithmic sensor is the same as that of an ordinary image sensor. If a main subject is clearly identifiable, an image of the main subject can be captured with a high contrast by controlling an exposure in such a manner that the optimal exposure level lies in a linear characteristic region. On the hand, in the logarithmic characteristic, a low contrast image may be outputted by photoelectric conversion in the logarithmic characteristic because an output in the logarithmic characteristic is compressed to one severalth of an output in the linear characteristic, despite that the logarithmic characteristic has an incident light intensity range one hundred times or more as wide as that in the linear characteristic. If the captured image is processed by an ordinary processing method, and the processed image is outputted to an output device such as a monitor or a printer as it is, the outputted image may have a low contrast in a high luminance region i.e. the logarithmic characteristic region, despite that the main subject image has a high contrast and the entirety of the image has a wide dynamic range. The linear-logarithmic sensor has another drawback that an extremely wide output dynamic range is required if the logarithmic characteristic is converted into the linear characteristic. As a result, it is impossible to output an image with such a wide dynamic range to an output device having a narrow dynamic range without processing an image. If the wide dynamic range is compressed in accordance with the narrow dynamic range of the output device without processing an image, the entirety of the output image may have a low contrast. In view of this, an image processing capable of outputting a wide dynamic range image to the output device with a high contrast is required.
The technique of converting a wide dynamic range image into a narrow dynamic range image is called a dynamic range compression. In the dynamic range compression, according to the Retinex theory, light incident onto the retina is defined by the product of an illumination component and a reflectance component with respect to an object. Visual perception has a strong correlation to the reflectance component. In other words, by exclusively narrowing the dynamic range of the illumination component in the wide dynamic range image, a compressed image with a high contrast and a narrow dynamic range can be obtained while preserving the reflectance component having a strong correlation to visual perception.
It is technically difficult to accurately separate an illumination component and a reflectance component in an image. In an ordinary dynamic range compression, frequency separation is often performed to separate the illumination component and the reflectance component. Generally, the illumination component has a spatial frequency that moderately changes, and has a low frequency as compared with the reflectance component. In view of this, the illumination component is extracted by using a low-pass filter (LPF). If the size of the LPF is small, for instance, a two-dimensional digital filter of 3×3 or 5×5 is used, the reflectance component may remain in the extracted illumination component. As a result, the reflectance component in the extracted illumination component may also be compressed in compressing the illumination component, which may lower the contrast in the entire image. In view of this, a relatively large-sized LPF of e.g. 50×50 is required.
If a relatively large-sized LPF i.e. a linear LPF or a Gaussian LPF of performing weighted averaging alone is used, a halo effect may occur at a site where the illumination component in the image sharply changes. The halo effect occurs due to inconsistency between the extracted illumination component and the real illumination component. Particularly, the halo effect is serious in an edge portion of the image. FIG. 12 is a conceptual diagram showing a dynamic range compression using a one-dimensional signal waveform. As shown in the upper part of the illustration E, the real illumination component in an input image I shown in the illustration A has a feature that an edge component is preserved and a component other than the edge component is smoothed. The lower part of the illustration E shows the real reflectance component. If the input image I is filtered through a linear LPF, as shown in the upper part of the illustration B, an illumination component L with a dull edge is extracted. If a reflectance component D is extracted based on the illumination component L and the input image I, the waveform shown in the lower part of the illustration B is obtained as the reflectance component D. The reflectance component D may be fluctuated on or around the edge portion. If the illumination component L is compressed to an illumination component L′ as shown in the illustration C, and the product of the compressed illumination component L′ and the reflectance component D is calculated, a dynamic range compressed image I′ is obtained. As shown in the illustration D, an overshoot region and an undershoot region i.e. halo effects indicated by the encircled portions are generated in the dynamic range compressed image I′. In order to eliminate the halo effect, an LPF capable of preserving edge information i.e. a nonlinear filter called an edge preserving filter is required in extracting an illumination component.
Known examples of the edge preserving filter are an epsilon filter disclosed in a technical document D1, and a bilateral filter disclosed in a technical document D2.    D1: “ε-Separating Nonlinear Digital Filter and its Application” Harashima et al., Institute of Electronics, Information and Communication Engineers (IEICE), Vol. J65-A, No. 4, pp. 297-304, Apr. 1982    D2: “Fast Bilateral Filtering for the Display of High-Dynamic-Range Images” Fredo Durand and Julie Dorsey, SIGGRAPH 2002.
FIG. 13 is a diagram for describing portions where a luminance difference is large in an image G having a dark portion and a bright portion, specifically, edge portions 901 and 902. A signal waveform in the upper part of FIG. 13 represents the edge portion 901 of an object with a slope. The linear filter performs filtering, merely considering a spatial weight. However, the edge preserving filter performs filtering, additionally considering a weight or a threshold value in a luminance direction i.e. vertical directions indicated by the arrows P. In other words, assuming that an LPF having a certain filter size is indicated by a rectangular block 910, although the linear filter merely changes the magnitude of the signal waveform in the spatial direction i.e. horizontal directions indicated by the arrows 911, the edge preserving filter is capable of changing the magnitude of the signal waveform in the luminance direction i.e. vertical directions indicated by arrows 912, as well as in the spatial direction.
Assuming that an epsilon filter is indicated by a rectangular block 920, the epsilon filter enables to smooth all the luminance values within the block 920 into values represented by a straight line 922 passing a targeted luminance value i.e. a targeted pixel value 921. A luminance value outside the block 920 e.g. a luminance value 923 is replaced by a luminance value 924 on the straight line 922. In use of the epsilon filter, the magnitude of the signal waveform in the luminance direction changes depending on the threshold value E. On the other hand, assuming that a bilateral filter is indicated by the same rectangular block 920, the magnitude of the signal waveform in the luminance direction changes by a weight of a Gaussian curve 925 having a characteristic that the weight is approximated to zero, as the luminance value is away from the targeted luminance value 921, in other words, by a weight in the luminance direction corresponding to the threshold value E. As mentioned above, the filter size of the edge preserving filter changes in the luminance direction by changing the threshold value or the weight in the luminance direction.
It is necessary to increase the smoothness of the illumination component in order to secure a high contrast in an output image. Specifically, smoothness is reduced if a reflectance component is left in the illumination component. In view of this, it is required to secure a certain filter size for the edge preserving filter. The edge portion 902 in the image G shows a certain site on a linear object or a rod object 903. In processing the edge portion 902, the following drawback should be considered. The edge portion should be extracted as a reflectance component. An output 931 of the edge preserving filter in the illustration 930 has a configuration that an apex portion of the output 931 is indented or cut away as shown by the solid line 932. The edge component as the reflectance component is left in the illumination component. In other words, part of a detail component is left in the illumination component. This may cause an edge defect such as a pseudo edge in the linear object or the rod object. In view of this, the weight or the threshold value in the luminance direction is increased, i.e. the weight level or the threshold level is increased to extract the edge component as the reflectance component as shown in the illustration 940. This operation, however, may generate a displacement from the actual luminance level as indicated by portions 951 in an edge portion other than the edge portion 902 e.g. the edge portion 901. In other words, since the entirety of the waveform is smoothed in the similar manner as in the case of using the linear LPF, a halo effect may occur in the edge portion 901 by dynamic range compression in the similar manner as described above.